No Free Lunch with the Sandwich

In nonlinear regression theory, the sandwich estimator of the covariance matrix of the model parameters is known as a consistent estimator, even when the parameterized model does not contain the regression. However, in the latter case, we emphasize the fact that the consistency of the sandwich holds only if the inputs of the training set are the values of independent identically distributed random variables. Thus, in the frequent practical modeling situation involving a training set whose inputs are deliberately chosen and imposed by the designer, we question the opportunity to use the sandwich estimator rather than the simple estimator based on the inverse squared Jacobian. Index Terms – confidence intervals, experimental design, fixed inputs, heteroscedasticiy, i.i.d. random inputs, i.n.i.d. random inputs, least squares estimation, linear Taylor expansion, model misspecification, neural networks, parameter covariance matrix, nonlinear regression, sandwich estimator. 1. Motivation In some statistical tests for the comparison between candidate neural models, for the estimation of a confidence interval for the conditional mean of the process output, or for the detection of outliers, an estimate of the covariance matrix of the network parameters is needed. Various estimators have been established, among them: a) the estimator based on the inverse squared Jacobian (ISJ), which is consistent if the parameterized model contains the regression and if the noise is homoscedastic; this consistency holds, be the input values of the training set fixed, or realizations of independent identically distributed (i.i.d.) random variables, or even realizations of independent not identically distributed (i.n.i.d.) random